Papers with multilingual language model
Graph-Linguistic Fusion: Using Language Models for Wikidata Vandalism Detection (2025.acl-industry)
Copied to clipboard
| Challenge: | Wikidata is a large open-source structured knowledge base that is used by search engines, robots, and scripts. |
| Approach: | They propose a vandalism detection system for Wikidata that converts edits into a single space using a method called Graph2Text. |
| Outcome: | The proposed system outperforms the current production system and is released under an open license. |
Learning Compact Metrics for MT (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in machine translation and multilingual text generation have led researchers to adopt trained metrics such as COMET or BLEURT, which treat evaluation as a regression problem and use representations from multilingual pre-trained models such as XLM-RoBERTa or mBERT. |
| Approach: | They propose to use multilingual model capacity to improve model performance by transferring knowledge from one teacher to multiple students trained on related languages. |
| Outcome: | The proposed model yields 10.5% improvement over vanilla fine-tuning and reaches 92.6% of RemBERT’s performance using only a third of its parameters. |
Cross-lingual Few-Shot Learning on Unseen Languages (2022.aacl-main)
Copied to clipboard
| Challenge: | Large pre-trained language models have demonstrated the ability to obtain good performance on downstream tasks with limited examples in resource-rich languages. |
| Approach: | They propose to use a downstream sentiment analysis task to analyze the effectiveness of several few-shot learning strategies across 12 languages, including 8 unseen languages, to compare results. |
| Outcome: | The proposed model, XLM-R, gives the best performance on a task with few examples in resource-rich languages. |
Language-agnostic BERT Sentence Embedding (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for learning bilingual sentence embeddings are not well explored. |
| Approach: | They propose to combine best methods for learning multilingual sentence embeddings with pre-trained models to achieve 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba. |
| Outcome: | The proposed model achieves 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, above the 65.5% achieved by LASER. |
Exploring Cross-Lingual Transfer Learning with Unsupervised Machine Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | a new CLTL model is proposed to facilitate cross-linguistic transfer learning between distant languages . a key to CLTL is to learn a shared representation space for the given source-target language pair. |
| Approach: | They propose a new CLTL model that integrates machine translation with MT . they use an unannotated data technique to make use of the model's pre-training and fine-tuning . |
| Outcome: | The proposed model achieves better CLTL performance than the baseline model without more annotated data. |
First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT (2021.eacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have demonstrated remarkable zero-shot cross-lingual transfer capabilities. |
| Approach: | They propose to use a layer ablation technique to create a multilingual model that is viewed as a stacking of two sub-networks: a language-agnostic encoder and a task-specific predictor. |
| Outcome: | The proposed model can perform zero-shot cross-lingual transfer for many languages. |
Struct-XLM: A Structure Discovery Multilingual Language Model for Enhancing Cross-lingual Transfer through Reinforcement Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods require syntactic labels that are difficult to obtain and of poor quality for low-resource languages. |
| Approach: | They propose a syntactic alignment model that leverages reinforcement learning to discover universal syntaktic structures for cross-lingual PLM alignment. |
| Outcome: | The proposed model improves cross-lingual representation alignment on the XTREME benchmark. |
PAELLA: Parameter-Efficient Lightweight Language-Agnostic Captioning Model (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing models that only generate English captions are expensive due to the trend of scaling both data and model size. |
| Approach: | They propose a parameter-efficient lightweight language-agnostic captioning model that uses retrieval enhancement to train parameters between a visual model and a multilingual language model. |
| Outcome: | The proposed model outperforms models with more parameters and data and shows strong zero-shot abilities in low-resource languages. |
Leveraging Knowledge in Multilingual Commonsense Reasoning (2022.findings-acl)
Copied to clipboard
| Challenge: | Commonsense reasoning is a language-agnostic process, but most comprehensive knowledge sources are limited to a small number of languages, especially English. |
| Approach: | They propose to use English as a pivot language to integrate commonsense reasoning into models using a translate-retrieve-translate strategy. |
| Outcome: | The proposed model outperforms the state-of-the-art on the XCSR benchmarks. |
How about Time? Probing a Multilingual Language Model for Temporal Relations (2022.coling-1)
Copied to clipboard
| Challenge: | XLM-R is a multilingual language model for temporal relation classification between events in four languages. |
| Approach: | They propose to use a multilingual language model for temporal relation classification between events in four languages to obtain contextualized embeddings. |
| Outcome: | The proposed model outperforms state-of-the-art models in obtaining competitive results against state- of-the art systems, but lacks suitable encoded information to address this task. |
Wiki-40B: Multilingual Language Model Dataset (2020.lrec-1)
Copied to clipboard
| Challenge: | We propose a new multilingual language model benchmark that is composed of 40+ languages spanning several scripts and linguistic families. |
| Approach: | They propose a multilingual language model benchmark composed of 40+ languages . they train monolingual causal language models using a state-of-the-art model . |
| Outcome: | The proposed model is composed of 40+ languages spanning several scripts and linguistic families. |
ZGUL: Zero-shot Generalization to Unseen Languages using Multi-source Ensembling of Language Adapters (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to zero-shot cross-lingual transfer have focused on training with adapters of a single source and testing either with the target LA or LA of another related language. |
| Approach: | They propose to leverage LAs of multiple (linguistically or geographically related) source languages for more effective cross-lingual transfer instead of just one source LA . they extend their novel neural architecture, ZGUL, to settings where either (1) some unlabeled data or (2) few-shot training examples are available for the target language . |
| Outcome: | Extensive experimentation across four language groups, covering 15 unseen target languages, shows improvements of up to 3.2 average F1 points over baselines on POS tagging and NER tasks. |
mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for improving multilingual models only use entity information in pretraining and do not explicitly use entities in downstream tasks. |
| Approach: | They propose to leverage Wikipedia entity representations for downstream tasks . they train a multilingual language model with 24 languages with entity representation . |
| Outcome: | The proposed model outperforms word-based models in cross-lingual transfer tasks. |
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting (2023.acl-long)
Copied to clipboard
Zheng Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Winata, Stella Biderman, Edward Raff, Dragomir Radev, Vassilina Nikoulina
| Challenge: | Existing language adaptation strategies for multilingual models are limited to 46 languages . a new language is added to the model to improve zero-shot prompting performance . |
| Approach: | They apply existing language adaptation strategies to BLOOM and benchmark its zero-shot prompting performance on eight new languages in a resource-constrained setting. |
| Outcome: | The proposed model can be extended to other languages without incurring prohibitively large costs. |
XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models (2023.emnlp-main)
Copied to clipboard
Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, Madian Khabsa
| Challenge: | Large multilingual models rely on a single vocabulary shared across 100+ languages . this vocabulary bottleneck limits the representational capabilities of multilingual model XLM-R . |
| Approach: | They propose a new approach for scaling to large multilingual vocabularies by de-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity to achieve sufficient coverage for each individual language. |
| Outcome: | The proposed model outperforms XLM-R on all language tasks and is particularly effective on low-resource tasks. |
ICU: Conquering Language Barriers in Vision-and-Language Modeling by Dividing the Tasks into Image Captioning and Language Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that use multilingual captions for images have limited results due to the scarcity of training data. |
| Approach: | They propose a multilingual vision-and-language model that divides a V&L task into two stages . they propose IC, which takes the caption as the alt text and performs cross-lingual language understanding . |
| Outcome: | The proposed model can achieve state-of-the-art results for five languages and comparable results for the rest. |
Multilingual Substitution-based Word Sense Induction (2024.lrec-main)
Copied to clipboard
| Challenge: | Word Sense Induction is the task of finding senses of an ambiguous word . many approaches to WSI are language-specific and are not easily adaptable to new languages. |
| Approach: | They propose to use multilingual substitution-based WSI methods that generalize to any language supported by the underlying multilingual language model with minimal to no adaptation required. |
| Outcome: | The proposed methods perform on par with monolingual approaches on popular English datasets while being language-specific. |